Estimation, Optimization, and Parallelism when Data is Sparse or Highly Varying

We study stochastic optimization problems when the data is sparse, which is in a sensedual to the current understanding of high-dimensional statistical learning and optimization.We highlight both the difficulties—in terms of increased sample complexity that sparse datanecessitates—and the potential benefits, in terms of allowing parallelism and asynchrony in thedesign of algorithms. Concretely, we derive matching upper and lower bounds on the minimaxrate for optimization and learning with sparse data, and we exhibit algorithms achieving theserates. We also show how leveraging sparsity leads to (still minimax optimal) parallel andasynchronous algorithms, providing experimental evidence complementing our theoretical resultson several medium to large-scale learning tasks.